2024-12-17 00:06:17,669 [ 39232 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy (runner:42, check_args_and_update_paths) 2024-12-17 00:06:17,670 [ 39232 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration (runner:90, check_args_and_update_paths) 2024-12-17 00:06:17,670 [ 39232 ] INFO : src dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/src (runner:97, check_args_and_update_paths) 2024-12-17 00:06:17,670 [ 39232 ] INFO : base_configs_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration (runner:99, check_args_and_update_paths) 2024-12-17 00:06:17,670 [ 39232 ] INFO : Unknown image altinityinfra/dotnet-client (runner:357, ) Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_0qpkhw --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_BASE_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_KERBERIZED_HADOOP_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_KERBEROS_KDC_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_MYSQL_JAVA_CLIENT_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_MYSQL_JS_CLIENT_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_MYSQL_PHP_CLIENT_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_NGINX_DAV_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_NEW_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo]' test_distributed_inter_server_secret/test.py::test_secure test_distributed_inter_server_secret/test.py::test_secure_disagree test_distributed_inter_server_secret/test.py::test_secure_disagree_insert test_distributed_inter_server_secret/test.py::test_secure_insert_async test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async test_distributed_inter_server_secret/test.py::test_secure_insert_sync 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[default-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[default-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[pass-foo]' test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority]' 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative]' test_distributed_load_balancing/test.py::test_load_balancing_round_robin test_dotnet_client/test.py::test_dotnet_client test_encrypted_disk_replication/test.py::test_replication test_encrypted_disk_replication/test.py::test_zero_copy_replication test_endpoint_macro_substitution/test.py::test_different_types test_endpoint_macro_substitution/test.py::test_select_by_type test_executable_user_defined_functions_config_reload/test.py::test test_globs_in_filepath/test.py::test_deep_structure test_globs_in_filepath/test.py::test_linear_structure test_globs_in_filepath/test.py::test_strange_filenames test_globs_in_filepath/test.py::test_table_function_and_virtual_columns test_graphite_merge_tree/test.py::test_broken_partial_rollup test_graphite_merge_tree/test.py::test_combined_rules test_graphite_merge_tree/test.py::test_combined_rules_with_default test_graphite_merge_tree/test.py::test_multiple_output_blocks test_graphite_merge_tree/test.py::test_multiple_paths_and_versions test_graphite_merge_tree/test.py::test_path_dangling_pointer test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern test_graphite_merge_tree/test.py::test_rollup_aggregation test_graphite_merge_tree/test.py::test_rollup_aggregation_2 test_graphite_merge_tree/test.py::test_rollup_versions test_graphite_merge_tree/test.py::test_system_graphite_retentions test_graphite_merge_tree/test.py::test_wrong_rollup_config test_grpc_protocol_ssl/test.py::test_insecure_channel test_grpc_protocol_ssl/test.py::test_secure_channel test_grpc_protocol_ssl/test.py::test_wrong_client_certificate test_hedged_requests/test.py::test_async_connect test_hedged_requests/test.py::test_async_query_sending test_hedged_requests/test.py::test_combination1 test_hedged_requests/test.py::test_combination2 test_hedged_requests/test.py::test_combination3 test_hedged_requests/test.py::test_combination4 test_hedged_requests/test.py::test_initial_receive_timeout test_hedged_requests/test.py::test_long_query test_hedged_requests/test.py::test_receive_timeout1 test_hedged_requests/test.py::test_receive_timeout2 test_hedged_requests/test.py::test_send_data test_hedged_requests/test.py::test_send_data2 test_hedged_requests/test.py::test_send_table_status_sleep test_hedged_requests/test.py::test_send_table_status_sleep2 test_hedged_requests/test.py::test_stuck_replica test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4 test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4_fails_with_wrong_resolution test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v6 test_http_handlers_config/test.py::test_absolute_path_static_handler test_http_handlers_config/test.py::test_config_static_handler test_http_handlers_config/test.py::test_defaults_http_handlers test_http_handlers_config/test.py::test_dynamic_query_handler test_http_handlers_config/test.py::test_fixed_static_handler test_http_handlers_config/test.py::test_predefined_query_handler test_http_handlers_config/test.py::test_prometheus_handler test_http_handlers_config/test.py::test_relative_path_static_handler test_http_handlers_config/test.py::test_replicas_status_handler test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[1]' test_jemalloc_percpu_arena/test.py::test_jemalloc_percpu_arena test_keeper_feature_flags_config/test.py::test_keeper_feature_flags -vvv" altinityinfra/integration-tests-runner:0-b53fc9f68f30219552c427c7d7d451342728b9a4 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: repeat-0.9.3, random-0.2, timeout-2.2.0, order-1.0.1, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] test_graphite_merge_tree/test.py::test_broken_partial_rollup test_globs_in_filepath/test.py::test_deep_structure test_http_handlers_config/test.py::test_absolute_path_static_handler test_grpc_protocol_ssl/test.py::test_insecure_channel test_hedged_requests/test.py::test_async_connect test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4 test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_encrypted_disk_replication/test.py::test_replication [gw1] [ 1%] PASSED test_graphite_merge_tree/test.py::test_broken_partial_rollup test_graphite_merge_tree/test.py::test_combined_rules [gw8] [ 2%] PASSED test_grpc_protocol_ssl/test.py::test_insecure_channel test_grpc_protocol_ssl/test.py::test_secure_channel [gw8] [ 3%] PASSED test_grpc_protocol_ssl/test.py::test_secure_channel test_grpc_protocol_ssl/test.py::test_wrong_client_certificate [gw1] [ 4%] PASSED test_graphite_merge_tree/test.py::test_combined_rules test_graphite_merge_tree/test.py::test_combined_rules_with_default [gw8] [ 5%] PASSED test_grpc_protocol_ssl/test.py::test_wrong_client_certificate [gw1] [ 6%] PASSED test_graphite_merge_tree/test.py::test_combined_rules_with_default [gw4] [ 7%] PASSED test_http_handlers_config/test.py::test_absolute_path_static_handler test_http_handlers_config/test.py::test_config_static_handler test_graphite_merge_tree/test.py::test_multiple_output_blocks [gw1] [ 8%] PASSED test_graphite_merge_tree/test.py::test_multiple_output_blocks test_graphite_merge_tree/test.py::test_multiple_paths_and_versions [gw1] [ 9%] PASSED test_graphite_merge_tree/test.py::test_multiple_paths_and_versions [gw2] [ 10%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[1] test_graphite_merge_tree/test.py::test_path_dangling_pointer test_dotnet_client/test.py::test_dotnet_client [gw2] [ 11%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[1] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[0] [gw5] [ 12%] FAILED test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default [gw3] [ 13%] PASSED test_hedged_requests/test.py::test_async_connect test_hedged_requests/test.py::test_async_query_sending [gw1] [ 14%] PASSED test_graphite_merge_tree/test.py::test_path_dangling_pointer test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern [gw1] [ 15%] PASSED test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern test_graphite_merge_tree/test.py::test_rollup_aggregation [gw2] [ 16%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[0] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[1] [gw1] [ 17%] PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation test_graphite_merge_tree/test.py::test_rollup_aggregation_2 [gw1] [ 18%] PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation_2 test_graphite_merge_tree/test.py::test_rollup_versions [gw9] [ 19%] PASSED test_encrypted_disk_replication/test.py::test_replication test_encrypted_disk_replication/test.py::test_zero_copy_replication [gw1] [ 20%] PASSED test_graphite_merge_tree/test.py::test_rollup_versions test_graphite_merge_tree/test.py::test_system_graphite_retentions [gw4] [ 21%] PASSED test_http_handlers_config/test.py::test_config_static_handler test_http_handlers_config/test.py::test_defaults_http_handlers [gw1] [ 22%] PASSED test_graphite_merge_tree/test.py::test_system_graphite_retentions [gw9] [ 23%] PASSED test_encrypted_disk_replication/test.py::test_zero_copy_replication test_graphite_merge_tree/test.py::test_wrong_rollup_config [gw1] [ 24%] PASSED test_graphite_merge_tree/test.py::test_wrong_rollup_config [gw2] [ 25%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[1] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[0] test_jemalloc_percpu_arena/test.py::test_jemalloc_percpu_arena [gw0] [ 26%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] [gw0] [ 27%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] [gw0] [ 28%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] [gw0] [ 29%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] [gw2] [ 30%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[0] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[1] [gw0] [ 31%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_secure [gw0] [ 32%] PASSED test_distributed_inter_server_secret/test.py::test_secure test_distributed_inter_server_secret/test.py::test_secure_disagree [gw0] [ 33%] PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree test_distributed_inter_server_secret/test.py::test_secure_disagree_insert [gw1] [ 34%] SKIPPED test_jemalloc_percpu_arena/test.py::test_jemalloc_percpu_arena test_keeper_feature_flags_config/test.py::test_keeper_feature_flags [gw0] [ 35%] PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree_insert test_distributed_inter_server_secret/test.py::test_secure_insert_async [gw0] [ 36%] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_async test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async [gw3] [ 37%] PASSED test_hedged_requests/test.py::test_async_query_sending test_hedged_requests/test.py::test_combination1 [gw2] [ 38%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[1] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[0] [gw0] [ 39%] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async test_distributed_inter_server_secret/test.py::test_secure_insert_sync [gw4] [ 40%] PASSED test_http_handlers_config/test.py::test_defaults_http_handlers test_http_handlers_config/test.py::test_dynamic_query_handler [gw0] [ 41%] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_sync test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] [gw0] [ 42%] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] [gw0] [ 43%] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] [gw2] [ 44%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[0] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[1] test_endpoint_macro_substitution/test.py::test_different_types [gw0] [ 45%] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] [gw3] [ 46%] PASSED test_hedged_requests/test.py::test_combination1 test_hedged_requests/test.py::test_combination2 [gw0] [ 47%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] [gw2] [ 48%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[1] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-0] [gw0] [ 49%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] [gw0] [ 50%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[default-] [gw0] [ 51%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[default-] test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[nopass-] [gw4] [ 52%] PASSED test_http_handlers_config/test.py::test_dynamic_query_handler test_http_handlers_config/test.py::test_fixed_static_handler [gw0] [ 53%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[nopass-] test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[pass-foo] [gw2] [ 54%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-0] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-1] [gw0] [ 55%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[pass-foo] test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[default-] [gw3] [ 56%] PASSED test_hedged_requests/test.py::test_combination2 test_hedged_requests/test.py::test_combination3 [gw0] [ 57%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[default-] test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[nopass-] [gw0] [ 58%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[nopass-] test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[pass-foo] [gw0] [ 59%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[pass-foo] [gw2] [ 60%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-1] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-0] [gw4] [ 61%] PASSED test_http_handlers_config/test.py::test_fixed_static_handler test_http_handlers_config/test.py::test_predefined_query_handler [gw2] [ 62%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-0] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-1] [gw3] [ 63%] PASSED test_hedged_requests/test.py::test_combination3 test_hedged_requests/test.py::test_combination4 [gw6] [ 64%] PASSED test_globs_in_filepath/test.py::test_deep_structure test_globs_in_filepath/test.py::test_linear_structure [gw2] [ 65%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-1] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[0] [gw8] [ 66%] PASSED test_dotnet_client/test.py::test_dotnet_client [gw1] [ 67%] PASSED test_keeper_feature_flags_config/test.py::test_keeper_feature_flags test_executable_user_defined_functions_config_reload/test.py::test [gw2] [ 68%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[0] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[1] [gw6] [ 69%] PASSED test_globs_in_filepath/test.py::test_linear_structure test_globs_in_filepath/test.py::test_strange_filenames [gw5] [ 70%] FAILED test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random [gw3] [ 71%] PASSED test_hedged_requests/test.py::test_combination4 test_hedged_requests/test.py::test_initial_receive_timeout [gw4] [ 72%] PASSED test_http_handlers_config/test.py::test_predefined_query_handler test_http_handlers_config/test.py::test_prometheus_handler [gw6] [ 73%] PASSED test_globs_in_filepath/test.py::test_strange_filenames test_globs_in_filepath/test.py::test_table_function_and_virtual_columns [gw6] [ 74%] PASSED test_globs_in_filepath/test.py::test_table_function_and_virtual_columns [gw2] [ 75%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[1] [gw7] [ 76%] PASSED test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4 test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4_fails_with_wrong_resolution [gw3] [ 77%] PASSED test_hedged_requests/test.py::test_initial_receive_timeout test_hedged_requests/test.py::test_long_query [gw4] [ 78%] PASSED test_http_handlers_config/test.py::test_prometheus_handler test_http_handlers_config/test.py::test_relative_path_static_handler [gw3] [ 79%] PASSED test_hedged_requests/test.py::test_long_query test_hedged_requests/test.py::test_receive_timeout1 [gw4] [ 80%] PASSED test_http_handlers_config/test.py::test_relative_path_static_handler test_http_handlers_config/test.py::test_replicas_status_handler [gw8] [ 81%] PASSED test_executable_user_defined_functions_config_reload/test.py::test test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total [gw3] [ 82%] PASSED test_hedged_requests/test.py::test_receive_timeout1 test_hedged_requests/test.py::test_receive_timeout2 [gw3] [ 83%] PASSED test_hedged_requests/test.py::test_receive_timeout2 test_hedged_requests/test.py::test_send_data [gw9] [ 84%] PASSED test_endpoint_macro_substitution/test.py::test_different_types test_endpoint_macro_substitution/test.py::test_select_by_type [gw4] [ 85%] PASSED test_http_handlers_config/test.py::test_replicas_status_handler [gw9] [ 86%] PASSED test_endpoint_macro_substitution/test.py::test_select_by_type [gw3] [ 87%] PASSED test_hedged_requests/test.py::test_send_data test_hedged_requests/test.py::test_send_data2 [gw3] [ 88%] PASSED test_hedged_requests/test.py::test_send_data2 test_hedged_requests/test.py::test_send_table_status_sleep [gw7] [ 89%] PASSED test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4_fails_with_wrong_resolution test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v6 [gw8] [ 90%] PASSED test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total [gw3] [ 91%] PASSED test_hedged_requests/test.py::test_send_table_status_sleep test_hedged_requests/test.py::test_send_table_status_sleep2 [gw5] [ 92%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_in_order [gw3] [ 93%] PASSED test_hedged_requests/test.py::test_send_table_status_sleep2 test_hedged_requests/test.py::test_stuck_replica [gw3] [ 94%] PASSED test_hedged_requests/test.py::test_stuck_replica [gw5] [ 95%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname [gw7] [ 96%] PASSED test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v6 [gw5] [ 97%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] [gw5] [ 98%] FAILED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] [gw5] [ 99%] FAILED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] test_distributed_load_balancing/test.py::test_load_balancing_round_robin [gw5] [100%] FAILED test_distributed_load_balancing/test.py::test_load_balancing_round_robin =================================== FAILURES =================================== _________________ test_distributed_replica_max_ignored_errors __________________ [gw5] linux -- Python 3.10.12 /usr/bin/python3 def test_distributed_replica_max_ignored_errors(): settings = { "use_hedged_requests": 0, "load_balancing": "in_order", "prefer_localhost_replica": 0, "connect_timeout": 2, "receive_timeout": 2, "send_timeout": 2, "idle_connection_timeout": 2, "tcp_keep_alive_timeout": 2, "distributed_replica_max_ignored_errors": 0, "distributed_replica_error_half_life": 60, } # initiate connection (if started only this test) n2.query("SELECT * FROM dist", settings=settings) cluster.pause_container("n1") # n1 paused -- skipping, and increment error_count for n1 # but the query succeeds, no need in query_and_get_error() n2.query("SELECT * FROM dist", settings=settings) # XXX: due to config reloading we need second time (sigh) n2.query("SELECT * FROM dist", settings=settings) # check error_count for n1 assert ( int( n2.query( """ SELECT errors_count FROM system.clusters WHERE cluster = 'replicas_cluster' AND host_name = 'n1' """, settings=settings, ) ) == 1 ) cluster.unpause_container("n1") # still n2 assert get_node(n2, settings=settings) == "n2" # now n1 settings["distributed_replica_max_ignored_errors"] = 1 > assert get_node(n2, settings=settings) == "n1" E AssertionError: assert 'n2' == 'n1' E - n1 E + n2 test_distributed_load_balancing/test.py:227: AssertionError ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ------------------------------ Captured log setup ------------------------------ 2024-12-17 00:06:22 [ 375 ] DEBUG : Command:['docker ps | wc -l'] (cluster.py:105, run_and_check) 2024-12-17 00:06:22 [ 375 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check) 2024-12-17 00:06:22 [ 375 ] DEBUG : No running containers (conftest.py:60, cleanup_environment) 2024-12-17 00:06:22 [ 375 ] DEBUG : Pruning Docker networks (conftest.py:62, cleanup_environment) 2024-12-17 00:06:22 [ 375 ] DEBUG : Command:['docker network prune --force'] (cluster.py:105, run_and_check) 2024-12-17 00:06:22 [ 375 ] DEBUG : Command:["sysctl net.ipv4.ip_local_port_range='55000 65535'"] (cluster.py:105, run_and_check) 2024-12-17 00:06:22 [ 375 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:113, run_and_check) 2024-12-17 00:06:22 [ 375 ] INFO : Running tests in /ClickHouse/tests/integration/test_distributed_load_balancing/test.py (cluster.py:2637, start) 2024-12-17 00:06:22 [ 375 ] DEBUG : Cluster start called. is_up=False (cluster.py:2644, start) 2024-12-17 00:06:22 [ 375 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 35 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: 0d36187d319f ID: 254f1316-ad07-4c34-95b7-0e014b574989 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-12-17 00:06:22 [ 375 ] DEBUG : Docker networks for project roottestdistributedloadbalancing are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-12-17 00:06:22 [ 375 ] DEBUG : Docker containers for project roottestdistributedloadbalancing are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-12-17 00:06:22 [ 375 ] DEBUG : Docker volumes for project roottestdistributedloadbalancing are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-12-17 00:06:22 [ 375 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-12-17 00:06:22 [ 375 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 35 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: 0d36187d319f ID: 254f1316-ad07-4c34-95b7-0e014b574989 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-12-17 00:06:22 [ 375 ] DEBUG : Docker networks for project roottestdistributedloadbalancing are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-12-17 00:06:22 [ 375 ] DEBUG : Docker containers for project roottestdistributedloadbalancing are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-12-17 00:06:22 [ 375 ] DEBUG : Docker volumes for project roottestdistributedloadbalancing are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-12-17 00:06:22 [ 375 ] DEBUG : Command:docker container list --all --filter name='^/roottestdistributedloadbalancing_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-12-17 00:06:22 [ 375 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-12-17 00:06:22 [ 375 ] DEBUG : No running containers for project: roottestdistributedloadbalancing (cluster.py:782, cleanup) 2024-12-17 00:06:22 [ 375 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-12-17 00:06:22 [ 375 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-12-17 00:06:22 [ 375 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-12-17 00:06:22 [ 375 ] DEBUG : Stderr:Error response from daemon: a prune operation is already running (cluster.py:115, run_and_check) 2024-12-17 00:06:22 [ 375 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) 2024-12-17 00:06:22 [ 375 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-12-17 00:06:22 [ 375 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-12-17 00:06:22 [ 375 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check) 2024-12-17 00:06:22 [ 375 ] DEBUG : Setup directory for instance: n1 (cluster.py:2657, start) 2024-12-17 00:06:22 [ 375 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/configs/config.d (cluster.py:4383, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/database (cluster.py:4400, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/logs (cluster.py:4411, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Setup directory for instance: n2 (cluster.py:2657, start) 2024-12-17 00:06:22 [ 375 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/configs/config.d (cluster.py:4383, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/database (cluster.py:4400, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/logs (cluster.py:4411, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Setup directory for instance: n3 (cluster.py:2657, start) 2024-12-17 00:06:22 [ 375 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/configs/config.d (cluster.py:4383, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/database (cluster.py:4400, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/logs (cluster.py:4411, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir) 2024-12-17 00:06:22 [ 375 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/.env (cluster.py:78, _create_env_file) 2024-12-17 00:06:22 [ 375 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-12-17 00:06:22 [ 375 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-12-17 00:06:22 [ 375 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-12-17 00:06:22 [ 375 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-12-17 00:06:22 [ 375 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:546, _make_request) 2024-12-17 00:06:22 [ 375 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/docker-compose.yml', 'pull'] (cluster.py:105, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n2 ... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n1 ... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n3 ... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n1 ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n3 ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n1 ... digest: sha256:849539f95a578f74cb... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n1 ... status: image is up to date for a... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n3 ... digest: sha256:849539f95a578f74cb... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n3 ... status: image is up to date for a... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n3 ... done (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n2 ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n2 ... digest: sha256:849539f95a578f74cb... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n2 ... status: image is up to date for a... (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : Stderr:Pulling n2 ... done (cluster.py:115, run_and_check) 2024-12-17 00:06:34 [ 375 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/.env --project-name roottestdistributedloadbalancing --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/docker-compose.yml --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/docker-compose.yml up -d --no-recreate') (cluster.py:2980, start) 2024-12-17 00:06:34 [ 375 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:105, run_and_check) 2024-12-17 00:06:36 [ 375 ] DEBUG : Stderr:Creating network "roottestdistributedloadbalancing_default" with the default driver (cluster.py:115, run_and_check) 2024-12-17 00:06:36 [ 375 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n2_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:06:36 [ 375 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n3_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:06:36 [ 375 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n1_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:06:36 [ 375 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n2_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:06:36 [ 375 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n1_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:06:36 [ 375 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n3_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:06:36 [ 375 ] DEBUG : ClickHouse instance created (cluster.py:2988, start) 2024-12-17 00:06:36 [ 375 ] DEBUG : get_instance_ip instance_name=n1 (cluster.py:1971, get_instance_ip) 2024-12-17 00:06:36 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:36 [ 375 ] DEBUG : Waiting for ClickHouse start in n1, ip: 172.16.3.3... (cluster.py:2995, start) 2024-12-17 00:06:36 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:36 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:36 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:36 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:36 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:36 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:36 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:36 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:37 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:37 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:37 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:37 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:37 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:37 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:37 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:37 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:37 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:37 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:38 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:38 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:38 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:38 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:38 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:38 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:38 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:38 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:38 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:39 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/f3abfa92bf65f31a032754b778b89139ce637e10804311ae10beb660a4f24c55/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:39 [ 375 ] DEBUG : ClickHouse n1 started (cluster.py:2999, start) 2024-12-17 00:06:39 [ 375 ] DEBUG : get_instance_ip instance_name=n2 (cluster.py:1971, get_instance_ip) 2024-12-17 00:06:39 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:39 [ 375 ] DEBUG : Waiting for ClickHouse start in n2, ip: 172.16.3.2... (cluster.py:2995, start) 2024-12-17 00:06:39 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:39 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/3f40caf44c48370c8b4ab3e1b5518ac0171f0957a20fcdcb0d07704be5be5486/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:39 [ 375 ] DEBUG : ClickHouse n2 started (cluster.py:2999, start) 2024-12-17 00:06:39 [ 375 ] DEBUG : get_instance_ip instance_name=n3 (cluster.py:1971, get_instance_ip) 2024-12-17 00:06:39 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:39 [ 375 ] DEBUG : Waiting for ClickHouse start in n3, ip: 172.16.3.4... (cluster.py:2995, start) 2024-12-17 00:06:39 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n3_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:39 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/e544d544e8d52c29957a40bfb40d52dd82ec062cec1a82cc61c0c48711b94adf/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:39 [ 375 ] DEBUG : http://localhost:None "GET /v1.42/containers/e544d544e8d52c29957a40bfb40d52dd82ec062cec1a82cc61c0c48711b94adf/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-12-17 00:06:39 [ 375 ] DEBUG : ClickHouse n3 started (cluster.py:2999, start) 2024-12-17 00:06:39 [ 375 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n1 (cluster.py:3423, query) 2024-12-17 00:06:39 [ 375 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n1 (cluster.py:3423, query) 2024-12-17 00:06:39 [ 375 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n1 (cluster.py:3423, query) 2024-12-17 00:06:39 [ 375 ] DEBUG : Executing query CREATE TABLE dist AS data Engine=Distributed( replicas_cluster, currentDatabase(), data) on n1 (cluster.py:3423, query) 2024-12-17 00:06:40 [ 375 ] DEBUG : Executing query CREATE TABLE dist_priority AS data Engine=Distributed( replicas_priority_cluster, currentDatabase(), data) on n1 (cluster.py:3423, query) 2024-12-17 00:06:40 [ 375 ] DEBUG : Executing query CREATE TABLE dist_priority_negative AS data Engine=Distributed( replicas_priority_negative_cluster, currentDatabase(), data) on n1 (cluster.py:3423, query) 2024-12-17 00:06:40 [ 375 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n2 (cluster.py:3423, query) 2024-12-17 00:06:40 [ 375 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n2 (cluster.py:3423, query) 2024-12-17 00:06:41 [ 375 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n2 (cluster.py:3423, query) 2024-12-17 00:06:41 [ 375 ] DEBUG : Executing query CREATE TABLE dist AS data Engine=Distributed( replicas_cluster, currentDatabase(), data) on n2 (cluster.py:3423, query) 2024-12-17 00:06:41 [ 375 ] DEBUG : Executing query CREATE TABLE dist_priority AS data Engine=Distributed( replicas_priority_cluster, currentDatabase(), data) on n2 (cluster.py:3423, query) 2024-12-17 00:06:41 [ 375 ] DEBUG : Executing query CREATE TABLE dist_priority_negative AS data Engine=Distributed( replicas_priority_negative_cluster, currentDatabase(), data) on n2 (cluster.py:3423, query) 2024-12-17 00:06:41 [ 375 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n3 (cluster.py:3423, query) 2024-12-17 00:06:42 [ 375 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n3 (cluster.py:3423, query) 2024-12-17 00:06:42 [ 375 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n3 (cluster.py:3423, query) 2024-12-17 00:06:42 [ 375 ] DEBUG : Executing query CREATE TABLE dist AS data Engine=Distributed( replicas_cluster, currentDatabase(), data) on n3 (cluster.py:3423, query) 2024-12-17 00:06:42 [ 375 ] DEBUG : Executing query CREATE TABLE dist_priority AS data Engine=Distributed( replicas_priority_cluster, currentDatabase(), data) on n3 (cluster.py:3423, query) 2024-12-17 00:06:43 [ 375 ] DEBUG : Executing query CREATE TABLE dist_priority_negative AS data Engine=Distributed( replicas_priority_negative_cluster, currentDatabase(), data) on n3 (cluster.py:3423, query) 2024-12-17 00:06:43 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n1 (cluster.py:3423, query) 2024-12-17 00:06:43 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n2 (cluster.py:3423, query) 2024-12-17 00:06:43 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n3 (cluster.py:3423, query) ------------------------------ Captured log call ------------------------------- 2024-12-17 00:06:44 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3423, query) 2024-12-17 00:06:44 [ 375 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/docker-compose.yml', 'pause', 'n1'] (cluster.py:105, run_and_check) 2024-12-17 00:06:44 [ 375 ] DEBUG : Stderr:Pausing roottestdistributedloadbalancing_n1_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:06:44 [ 375 ] DEBUG : Stderr:Pausing roottestdistributedloadbalancing_n1_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:06:44 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3423, query) 2024-12-17 00:06:47 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3423, query) 2024-12-17 00:06:47 [ 375 ] DEBUG : Executing query SELECT errors_count FROM system.clusters WHERE cluster = 'replicas_cluster' AND host_name = 'n1' on n2 (cluster.py:3423, query) 2024-12-17 00:06:47 [ 375 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/docker-compose.yml', 'unpause', 'n1'] (cluster.py:105, run_and_check) 2024-12-17 00:06:48 [ 375 ] DEBUG : Stderr:Unpausing roottestdistributedloadbalancing_n1_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:06:48 [ 375 ] DEBUG : Stderr:Unpausing roottestdistributedloadbalancing_n1_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:06:48 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3423, query) 2024-12-17 00:06:48 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:06:49 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:06:50 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:06:51 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '234030843dca4a4ab1f5d0178930ae55' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n2 (cluster.py:3423, query) 2024-12-17 00:06:51 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3423, query) 2024-12-17 00:06:52 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:06:52 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:06:53 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:06:54 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'a5edaf3acd5d419caaf33c5866ee7efc' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n2 (cluster.py:3423, query) _________________________ test_load_balancing_default __________________________ [gw5] linux -- Python 3.10.12 /usr/bin/python3 def test_load_balancing_default(): unique_nodes = set() for _ in range(0, queries): unique_nodes.add(get_node(n1, settings={"load_balancing": "random"})) > assert len(unique_nodes) == nodes, unique_nodes E AssertionError: {'n1'} E assert 1 == 3 E + where 1 = len({'n1'}) test_distributed_load_balancing/test.py:132: AssertionError ------------------------------ Captured log setup ------------------------------ 2024-12-17 00:06:54 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n1 (cluster.py:3423, query) 2024-12-17 00:06:54 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n2 (cluster.py:3423, query) 2024-12-17 00:06:55 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n3 (cluster.py:3423, query) ------------------------------ Captured log call ------------------------------- 2024-12-17 00:06:55 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:06:55 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:06:56 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:06:56 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:06:57 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '431b3d720d4e499fbfc959e44ecc023f' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:06:57 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:06:58 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:06:59 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:06:59 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:00 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'ccbc118222ba4f40aa9f05a4bc240529' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:00 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:00 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:01 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:02 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:02 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '7672e609809d4c9aa30bdbe91b84b5a2' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:03 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:03 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:04 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:04 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:05 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'bf065600194c483bb1409454985ea649' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:05 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:05 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:06 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:07 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:07 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '4e02b8454027481b8f2a25d54a6c1b4f' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:08 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:08 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:09 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:09 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:10 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '2d3cf764569b4c6f93a8eb0ec265e579' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:10 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:12 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:12 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:12 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:13 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'ed420548c8ec480ca5f950b06614bae5' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:13 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:14 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:14 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:15 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:15 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '804093b568ac4bec811113ac063dbf23' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:15 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:16 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:16 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:17 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:17 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'fa327ecdea8a4ebaacdf0cce931aaab7' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:18 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:18 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:18 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:19 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:19 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '61bcbff1a1534084ac020bf1fb3e92bd' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:20 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:20 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:21 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:21 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:22 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '37bcd986fa0643ffb4b02b1d62957371' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:22 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:22 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:23 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:23 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:24 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'f0990d24c1594391a06fbdb6e01748db' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:24 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:24 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:25 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:25 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:26 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '283e512978334ec4b3b4bba279a10683' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:26 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:26 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:27 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:28 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:28 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'd0a79320f4c446509c2c5bceb0190daa' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:28 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:29 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:29 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:30 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:30 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '39e3a13b42434021846216f6c2c96c1a' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:31 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:31 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:32 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:32 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:33 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'ddca161a631e4e1d9c55591c1ee86c6f' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:33 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:33 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:34 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:34 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:35 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'eba7f27b71244697b494162ff9434e44' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:35 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:36 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:36 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:37 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:37 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '284dd1a05fb341c6a473e75c615dd6cd' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:38 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:38 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:38 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:39 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:40 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '1ce4452d10064b618108ea13d02561d5' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:40 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:40 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:41 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:41 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:42 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '42503c3163a24cf7acbeb298033096e6' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:42 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:43 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:43 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:44 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:44 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'b70bb1093c22430889da6c3552e4be79' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:45 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:45 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:46 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:46 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:47 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '12743c8b36be47bdbd21e3deb6013fb2' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:47 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:47 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:48 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:49 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:49 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '4466083db6d54950ace17a63abcaf3fd' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:50 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:50 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:51 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:52 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:53 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '994931cf63424611a6f52768121a0c84' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:53 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:53 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:54 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:55 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:55 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '00a63cc15ad44620b012358ccbc642f6' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:56 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:56 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:57 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:07:57 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:07:58 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '3ff902e3fcaa42c0b97e69e18e0edb14' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:07:58 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:07:59 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:07:59 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:08:00 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:08:01 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '03f71d22142745ba98d35e903d350e5d' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:08:01 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:08:01 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:08:02 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:08:03 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:08:04 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'ecb68a37d6424485bcac28e83d1d47e0' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:08:04 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:08:05 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:08:06 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:08:06 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:08:07 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '118234089f8f42c2ab23477d78e961f5' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:08:08 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:08:08 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:08:09 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:08:09 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:08:10 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '387c23cb128c441989a57d0f3e651f11' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) ___________ test_load_balancing_priority_round_robin[dist_priority] ____________ [gw5] linux -- Python 3.10.12 /usr/bin/python3 dist_table = 'dist_priority' @pytest.mark.parametrize( "dist_table", [ ("dist_priority"), ("dist_priority_negative"), ], ) def test_load_balancing_priority_round_robin(dist_table): unique_nodes = set() for _ in range(0, nodes): unique_nodes.add( get_node(n1, dist_table, settings={"load_balancing": "round_robin"}) ) > assert len(unique_nodes) == 2, unique_nodes E AssertionError: {'n1'} E assert 1 == 2 E + where 1 = len({'n1'}) test_distributed_load_balancing/test.py:180: AssertionError ------------------------------ Captured log setup ------------------------------ 2024-12-17 00:12:19 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n1 (cluster.py:3423, query) 2024-12-17 00:12:20 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n2 (cluster.py:3423, query) 2024-12-17 00:12:20 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n3 (cluster.py:3423, query) ------------------------------ Captured log call ------------------------------- 2024-12-17 00:12:20 [ 375 ] DEBUG : Executing query SELECT * FROM dist_priority on n1 (cluster.py:3423, query) 2024-12-17 00:12:20 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:12:21 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:12:21 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:12:22 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '1f7436c3df0c4c79ad7d0ae0aa7c9077' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:12:22 [ 375 ] DEBUG : Executing query SELECT * FROM dist_priority on n1 (cluster.py:3423, query) 2024-12-17 00:12:23 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:12:23 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:12:24 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:12:24 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = 'ac5f20f5577a411bad5f27feab341187' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:12:25 [ 375 ] DEBUG : Executing query SELECT * FROM dist_priority on n1 (cluster.py:3423, query) 2024-12-17 00:12:25 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:12:25 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:12:26 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:12:26 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '44c6e014de98483eb14b59966f4d4825' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) _______ test_load_balancing_priority_round_robin[dist_priority_negative] _______ [gw5] linux -- Python 3.10.12 /usr/bin/python3 dist_table = 'dist_priority_negative' @pytest.mark.parametrize( "dist_table", [ ("dist_priority"), ("dist_priority_negative"), ], ) def test_load_balancing_priority_round_robin(dist_table): unique_nodes = set() for _ in range(0, nodes): unique_nodes.add( get_node(n1, dist_table, settings={"load_balancing": "round_robin"}) ) > assert len(unique_nodes) == 2, unique_nodes E AssertionError: {'n1'} E assert 1 == 2 E + where 1 = len({'n1'}) test_distributed_load_balancing/test.py:180: AssertionError ------------------------------ Captured log setup ------------------------------ 2024-12-17 00:12:27 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n1 (cluster.py:3423, query) 2024-12-17 00:12:27 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n2 (cluster.py:3423, query) 2024-12-17 00:12:27 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n3 (cluster.py:3423, query) ------------------------------ Captured log call ------------------------------- 2024-12-17 00:12:28 [ 375 ] DEBUG : Executing query SELECT * FROM dist_priority_negative on n1 (cluster.py:3423, query) 2024-12-17 00:12:28 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:12:29 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:12:29 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:12:30 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '1851e7271d074676bde246b7d7911a10' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:12:30 [ 375 ] DEBUG : Executing query SELECT * FROM dist_priority_negative on n1 (cluster.py:3423, query) 2024-12-17 00:12:30 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:12:31 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:12:32 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:12:32 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '70252a43d2a74e3e8981328a00fc6998' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:12:32 [ 375 ] DEBUG : Executing query SELECT * FROM dist_priority_negative on n1 (cluster.py:3423, query) 2024-12-17 00:12:33 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:12:33 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:12:34 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:12:34 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '6da2111aeaf1419dad3152d116e81e92' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) _______________________ test_load_balancing_round_robin ________________________ [gw5] linux -- Python 3.10.12 /usr/bin/python3 def test_load_balancing_round_robin(): unique_nodes = set() for _ in range(0, nodes): unique_nodes.add(get_node(n1, settings={"load_balancing": "round_robin"})) > assert len(unique_nodes) == nodes, unique_nodes E AssertionError: {'n1'} E assert 1 == 3 E + where 1 = len({'n1'}) test_distributed_load_balancing/test.py:163: AssertionError ------------------------------ Captured log setup ------------------------------ 2024-12-17 00:12:35 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n1 (cluster.py:3423, query) 2024-12-17 00:12:35 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n2 (cluster.py:3423, query) 2024-12-17 00:12:35 [ 375 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n3 (cluster.py:3423, query) ------------------------------ Captured log call ------------------------------- 2024-12-17 00:12:35 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:12:36 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:12:36 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:12:37 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:12:37 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '27a51d7be92949c8b4f0c441763eb7c2' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:12:38 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:12:38 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:12:39 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:12:39 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:12:40 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '3e51f8f7cd8b42c69d1e3c8a81b84382' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) 2024-12-17 00:12:40 [ 375 ] DEBUG : Executing query SELECT * FROM dist on n1 (cluster.py:3423, query) 2024-12-17 00:12:40 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3423, query) 2024-12-17 00:12:41 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3423, query) 2024-12-17 00:12:42 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3423, query) 2024-12-17 00:12:43 [ 375 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '13a7a3fc3a1347e286373ac1e4f6403a' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n1 (cluster.py:3423, query) --------------------------- Captured stderr teardown --------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ---------------------------- Captured log teardown ----------------------------- 2024-12-17 00:12:44 [ 375 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n1_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n3_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n2_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n1_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n2_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n3_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0/n3/docker-compose.yml', 'down', '--volumes'] (cluster.py:105, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n1_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n3_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n2_1 ... (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n1_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n2_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n3_1 ... done (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Stderr:Removing network roottestdistributedloadbalancing_default (cluster.py:115, run_and_check) 2024-12-17 00:12:51 [ 375 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-12-17 00:12:52 [ 375 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 37 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: 0d36187d319f ID: 254f1316-ad07-4c34-95b7-0e014b574989 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-12-17 00:12:52 [ 375 ] DEBUG : Docker networks for project roottestdistributedloadbalancing are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-12-17 00:12:52 [ 375 ] DEBUG : Docker containers for project roottestdistributedloadbalancing are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-12-17 00:12:52 [ 375 ] DEBUG : Docker volumes for project roottestdistributedloadbalancing are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-12-17 00:12:52 [ 375 ] DEBUG : Command:docker container list --all --filter name='^/roottestdistributedloadbalancing_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-12-17 00:12:52 [ 375 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-12-17 00:12:52 [ 375 ] DEBUG : No running containers for project: roottestdistributedloadbalancing (cluster.py:782, cleanup) 2024-12-17 00:12:52 [ 375 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-12-17 00:12:52 [ 375 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-12-17 00:12:52 [ 375 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-12-17 00:12:52 [ 375 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-12-17 00:12:52 [ 375 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-12-17 00:12:52 [ 375 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-12-17 00:12:52 [ 375 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-12-17 00:12:52 [ 375 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check) ============================== slowest durations =============================== 108.29s call test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 106.09s setup test_endpoint_macro_substitution/test.py::test_different_types 90.94s call test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4 90.94s call test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4_fails_with_wrong_resolution 90.51s call test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v6 81.06s call test_globs_in_filepath/test.py::test_deep_structure 75.74s call test_distributed_load_balancing/test.py::test_load_balancing_default 70.13s call test_distributed_load_balancing/test.py::test_load_balancing_in_order 67.05s call test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 63.85s call test_dotnet_client/test.py::test_dotnet_client 48.34s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] 44.65s call test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 37.91s setup test_encrypted_disk_replication/test.py::test_replication 32.63s call test_keeper_feature_flags_config/test.py::test_keeper_feature_flags 29.59s setup test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4 24.62s call test_hedged_requests/test.py::test_async_query_sending 24.48s teardown test_encrypted_disk_replication/test.py::test_zero_copy_replication 24.43s teardown test_endpoint_macro_substitution/test.py::test_select_by_type 22.83s setup test_executable_user_defined_functions_config_reload/test.py::test 22.47s call test_http_handlers_config/test.py::test_replicas_status_handler 22.46s call test_http_handlers_config/test.py::test_relative_path_static_handler 22.45s call test_http_handlers_config/test.py::test_predefined_query_handler 22.17s call test_executable_user_defined_functions_config_reload/test.py::test 22.02s setup test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 21.41s call test_http_handlers_config/test.py::test_absolute_path_static_handler 20.89s call test_http_handlers_config/test.py::test_config_static_handler 20.03s setup test_hedged_requests/test.py::test_async_connect 19.94s call test_http_handlers_config/test.py::test_defaults_http_handlers 19.00s call test_http_handlers_config/test.py::test_prometheus_handler 18.74s call test_hedged_requests/test.py::test_initial_receive_timeout 17.51s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] 17.50s setup test_globs_in_filepath/test.py::test_deep_structure 17.07s setup test_graphite_merge_tree/test.py::test_broken_partial_rollup 16.92s setup test_grpc_protocol_ssl/test.py::test_insecure_channel 16.84s call test_hedged_requests/test.py::test_long_query 15.29s setup test_keeper_feature_flags_config/test.py::test_keeper_feature_flags 15.22s call test_hedged_requests/test.py::test_send_data2 15.18s call test_hedged_requests/test.py::test_receive_timeout2 14.92s call test_hedged_requests/test.py::test_async_connect 14.87s call test_hedged_requests/test.py::test_combination3 14.79s call test_http_handlers_config/test.py::test_dynamic_query_handler 13.95s call test_hedged_requests/test.py::test_send_data 13.61s call test_hedged_requests/test.py::test_combination4 13.60s call test_hedged_requests/test.py::test_receive_timeout1 12.89s call test_hedged_requests/test.py::test_send_table_status_sleep 12.69s setup test_dotnet_client/test.py::test_dotnet_client 12.31s setup test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 11.92s call test_graphite_merge_tree/test.py::test_path_dangling_pointer 11.49s call test_http_handlers_config/test.py::test_fixed_static_handler 10.81s call test_hedged_requests/test.py::test_combination1 10.64s call test_hedged_requests/test.py::test_combination2 10.57s call test_hedged_requests/test.py::test_stuck_replica 10.16s call test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 10.14s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[pass-foo] 9.86s call test_globs_in_filepath/test.py::test_linear_structure 9.43s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[1] 8.67s teardown test_distributed_load_balancing/test.py::test_load_balancing_round_robin 8.56s call test_hedged_requests/test.py::test_send_table_status_sleep2 8.55s teardown test_hedged_requests/test.py::test_stuck_replica 8.00s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[0] 7.97s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[0] 7.65s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-0] 7.56s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[1] 7.51s call test_distributed_load_balancing/test.py::test_load_balancing_round_robin 7.44s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[0] 7.43s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[1] 7.43s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-1] 7.24s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[1] 6.95s call test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 6.88s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[1] 6.84s teardown test_globs_in_filepath/test.py::test_table_function_and_virtual_columns 6.70s call test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 6.54s call test_jemalloc_percpu_arena/test.py::test_jemalloc_percpu_arena 6.34s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-0] 6.24s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-1] 6.19s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] 5.85s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[0] 5.56s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[1] 5.54s teardown test_dotnet_client/test.py::test_dotnet_client 5.03s teardown test_graphite_merge_tree/test.py::test_wrong_rollup_config 4.64s teardown test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 4.27s teardown test_grpc_protocol_ssl/test.py::test_wrong_client_certificate 4.03s teardown test_executable_user_defined_functions_config_reload/test.py::test 3.70s teardown test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v6 3.44s call test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async 3.23s call test_globs_in_filepath/test.py::test_strange_filenames 3.18s teardown test_keeper_feature_flags_config/test.py::test_keeper_feature_flags 2.74s call test_globs_in_filepath/test.py::test_table_function_and_virtual_columns 2.46s call test_encrypted_disk_replication/test.py::test_replication 2.37s call test_encrypted_disk_replication/test.py::test_zero_copy_replication 2.11s call test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] 2.05s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] 2.03s call test_grpc_protocol_ssl/test.py::test_insecure_channel 2.00s call test_grpc_protocol_ssl/test.py::test_wrong_client_certificate 2.00s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] 1.95s call test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] 1.93s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] 1.92s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[pass-foo] 1.92s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[default-] 1.88s call test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] 1.87s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] 1.74s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[default-] 1.68s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[nopass-] 1.58s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[pass-foo] 1.53s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[nopass-] 1.48s call test_endpoint_macro_substitution/test.py::test_select_by_type 1.43s call test_distributed_inter_server_secret/test.py::test_secure_insert_async 1.38s call test_graphite_merge_tree/test.py::test_combined_rules_with_default 1.32s call test_graphite_merge_tree/test.py::test_rollup_versions 1.27s call test_distributed_inter_server_secret/test.py::test_secure_disagree_insert 1.22s call test_distributed_inter_server_secret/test.py::test_secure_insert_sync 1.10s setup test_distributed_load_balancing/test.py::test_load_balancing_in_order 1.07s call test_graphite_merge_tree/test.py::test_combined_rules 1.05s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] 1.05s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] 1.00s call test_graphite_merge_tree/test.py::test_system_graphite_retentions 1.00s setup test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 1.00s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] 0.96s setup test_distributed_load_balancing/test.py::test_load_balancing_default 0.95s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] 0.90s setup test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 0.90s setup test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 0.90s setup test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 0.81s call test_graphite_merge_tree/test.py::test_broken_partial_rollup 0.80s setup test_distributed_load_balancing/test.py::test_load_balancing_round_robin 0.75s call test_graphite_merge_tree/test.py::test_rollup_aggregation 0.74s call test_graphite_merge_tree/test.py::test_multiple_output_blocks 0.72s teardown test_encrypted_disk_replication/test.py::test_replication 0.65s call test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern 0.52s teardown test_graphite_merge_tree/test.py::test_path_dangling_pointer 0.42s call test_endpoint_macro_substitution/test.py::test_different_types 0.38s setup test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern 0.37s teardown test_graphite_merge_tree/test.py::test_multiple_paths_and_versions 0.37s call test_graphite_merge_tree/test.py::test_multiple_paths_and_versions 0.37s call test_graphite_merge_tree/test.py::test_wrong_rollup_config 0.37s call test_graphite_merge_tree/test.py::test_rollup_aggregation_2 0.37s call test_distributed_inter_server_secret/test.py::test_secure_disagree 0.32s setup test_graphite_merge_tree/test.py::test_path_dangling_pointer 0.27s teardown test_graphite_merge_tree/test.py::test_rollup_aggregation 0.27s teardown test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern 0.27s setup test_graphite_merge_tree/test.py::test_rollup_aggregation 0.27s setup test_graphite_merge_tree/test.py::test_rollup_versions 0.27s setup test_graphite_merge_tree/test.py::test_system_graphite_retentions 0.27s setup test_graphite_merge_tree/test.py::test_rollup_aggregation_2 0.27s setup test_graphite_merge_tree/test.py::test_multiple_paths_and_versions 0.27s setup test_graphite_merge_tree/test.py::test_multiple_output_blocks 0.27s teardown test_graphite_merge_tree/test.py::test_system_graphite_retentions 0.27s setup test_graphite_merge_tree/test.py::test_combined_rules_with_default 0.27s teardown test_graphite_merge_tree/test.py::test_rollup_versions 0.27s setup test_graphite_merge_tree/test.py::test_combined_rules 0.27s teardown test_graphite_merge_tree/test.py::test_rollup_aggregation_2 0.27s teardown test_graphite_merge_tree/test.py::test_multiple_output_blocks 0.27s call test_distributed_inter_server_secret/test.py::test_secure 0.27s teardown test_graphite_merge_tree/test.py::test_combined_rules_with_default 0.24s call test_grpc_protocol_ssl/test.py::test_secure_channel 0.22s setup test_graphite_merge_tree/test.py::test_wrong_rollup_config 0.22s teardown test_graphite_merge_tree/test.py::test_combined_rules 0.22s teardown test_graphite_merge_tree/test.py::test_broken_partial_rollup 0.04s setup test_http_handlers_config/test.py::test_absolute_path_static_handler 0.01s teardown test_endpoint_macro_substitution/test.py::test_different_types 0.00s teardown test_http_handlers_config/test.py::test_replicas_status_handler 0.00s teardown test_globs_in_filepath/test.py::test_deep_structure 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[default-] 0.00s teardown test_http_handlers_config/test.py::test_relative_path_static_handler 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] 0.00s setup test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4_fails_with_wrong_resolution 0.00s teardown test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4 0.00s teardown test_hedged_requests/test.py::test_receive_timeout1 0.00s setup test_http_handlers_config/test.py::test_replicas_status_handler 0.00s teardown test_hedged_requests/test.py::test_initial_receive_timeout 0.00s teardown test_http_handlers_config/test.py::test_prometheus_handler 0.00s setup test_endpoint_macro_substitution/test.py::test_select_by_type 0.00s teardown test_http_handlers_config/test.py::test_config_static_handler 0.00s setup test_jemalloc_percpu_arena/test.py::test_jemalloc_percpu_arena 0.00s teardown test_http_handlers_config/test.py::test_fixed_static_handler 0.00s teardown test_hedged_requests/test.py::test_async_connect 0.00s teardown test_hedged_requests/test.py::test_long_query 0.00s teardown test_hedged_requests/test.py::test_send_table_status_sleep 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[nopass-] 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[0] 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[default-] 0.00s teardown test_hedged_requests/test.py::test_receive_timeout2 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[0] 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-1] 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[nopass-] 0.00s setup test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] 0.00s teardown test_http_handlers_config/test.py::test_defaults_http_handlers 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[0] 0.00s setup test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v6 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 0.00s setup test_grpc_protocol_ssl/test.py::test_secure_channel 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 0.00s setup test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[1] 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] 0.00s setup test_http_handlers_config/test.py::test_defaults_http_handlers 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-0] 0.00s setup test_hedged_requests/test.py::test_combination1 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_default 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[0] 0.00s teardown test_hedged_requests/test.py::test_async_query_sending 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] 0.00s teardown test_http_handlers_config/test.py::test_predefined_query_handler 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-1] 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] 0.00s teardown test_hedged_requests/test.py::test_combination1 0.00s teardown test_hedged_requests/test.py::test_send_data 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[1] 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[pass-foo] 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-1] 0.00s setup test_hedged_requests/test.py::test_send_data2 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[1] 0.00s setup test_hedged_requests/test.py::test_send_table_status_sleep 0.00s setup test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[1] 0.00s setup test_http_handlers_config/test.py::test_predefined_query_handler 0.00s setup test_grpc_protocol_ssl/test.py::test_wrong_client_certificate 0.00s setup test_hedged_requests/test.py::test_send_table_status_sleep2 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[0] 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[1] 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[1] 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-0] 0.00s teardown test_hedged_requests/test.py::test_send_data2 0.00s setup test_hedged_requests/test.py::test_receive_timeout1 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[1] 0.00s setup test_hedged_requests/test.py::test_long_query 0.00s teardown test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-0] 0.00s teardown test_http_handlers_config/test.py::test_absolute_path_static_handler 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_insert_async 0.00s setup test_hedged_requests/test.py::test_send_data 0.00s setup test_globs_in_filepath/test.py::test_strange_filenames 0.00s teardown test_hedged_requests/test.py::test_combination3 0.00s setup test_hedged_requests/test.py::test_receive_timeout2 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-1] 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[default-] 0.00s setup test_hedged_requests/test.py::test_async_query_sending 0.00s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[1] 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[0] 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_disagree_insert 0.00s setup test_encrypted_disk_replication/test.py::test_zero_copy_replication 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] 0.00s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[pass-foo] 0.00s teardown test_jemalloc_percpu_arena/test.py::test_jemalloc_percpu_arena 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-0] 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[0] 0.00s teardown test_grpc_protocol_ssl/test.py::test_insecure_channel 0.00s teardown test_http_handlers_config/test.py::test_dynamic_query_handler 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_insert_sync 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] 0.00s teardown test_hedged_requests/test.py::test_send_table_status_sleep2 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[default-] 0.00s setup test_hedged_requests/test.py::test_combination3 0.00s setup test_globs_in_filepath/test.py::test_table_function_and_virtual_columns 0.00s teardown test_hedged_requests/test.py::test_combination4 0.00s teardown test_hedged_requests/test.py::test_combination2 0.00s teardown test_globs_in_filepath/test.py::test_linear_structure 0.00s setup test_hedged_requests/test.py::test_combination2 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_in_order 0.00s setup test_http_handlers_config/test.py::test_relative_path_static_handler 0.00s setup test_globs_in_filepath/test.py::test_linear_structure 0.00s setup test_hedged_requests/test.py::test_combination4 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[nopass-] 0.00s teardown test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4_fails_with_wrong_resolution 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_disagree_insert 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] 0.00s setup test_http_handlers_config/test.py::test_prometheus_handler 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_disagree 0.00s setup test_http_handlers_config/test.py::test_dynamic_query_handler 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[1] 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] 0.00s setup test_distributed_inter_server_secret/test.py::test_secure 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[pass-foo] 0.00s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[0] 0.00s teardown test_grpc_protocol_ssl/test.py::test_secure_channel 0.00s setup test_hedged_requests/test.py::test_stuck_replica 0.00s setup test_http_handlers_config/test.py::test_fixed_static_handler 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_insert_sync 0.00s setup test_http_handlers_config/test.py::test_config_static_handler 0.00s setup test_hedged_requests/test.py::test_initial_receive_timeout 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_insert_async 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_disagree 0.00s teardown test_globs_in_filepath/test.py::test_strange_filenames 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[nopass-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure =========================== short test summary info ============================ FAILED test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors FAILED test_distributed_load_balancing/test.py::test_load_balancing_default FAILED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] FAILED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] FAILED test_distributed_load_balancing/test.py::test_load_balancing_round_robin PASSED test_graphite_merge_tree/test.py::test_broken_partial_rollup PASSED test_grpc_protocol_ssl/test.py::test_insecure_channel PASSED test_grpc_protocol_ssl/test.py::test_secure_channel PASSED test_graphite_merge_tree/test.py::test_combined_rules PASSED test_grpc_protocol_ssl/test.py::test_wrong_client_certificate PASSED test_graphite_merge_tree/test.py::test_combined_rules_with_default PASSED test_http_handlers_config/test.py::test_absolute_path_static_handler PASSED test_graphite_merge_tree/test.py::test_multiple_output_blocks PASSED test_graphite_merge_tree/test.py::test_multiple_paths_and_versions PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[1] PASSED test_hedged_requests/test.py::test_async_connect PASSED test_graphite_merge_tree/test.py::test_path_dangling_pointer PASSED test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[0] PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation_2 PASSED test_encrypted_disk_replication/test.py::test_replication PASSED test_graphite_merge_tree/test.py::test_rollup_versions PASSED test_http_handlers_config/test.py::test_config_static_handler PASSED test_graphite_merge_tree/test.py::test_system_graphite_retentions PASSED test_encrypted_disk_replication/test.py::test_zero_copy_replication PASSED test_graphite_merge_tree/test.py::test_wrong_rollup_config PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[1] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[0] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] PASSED test_distributed_inter_server_secret/test.py::test_secure PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree_insert PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_async PASSED test_hedged_requests/test.py::test_async_query_sending PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[1] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async PASSED test_http_handlers_config/test.py::test_defaults_http_handlers PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_sync PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[0] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] PASSED test_hedged_requests/test.py::test_combination1 PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[1] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[default-] PASSED test_http_handlers_config/test.py::test_dynamic_query_handler PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[nopass-] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-0] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[pass-foo] PASSED test_hedged_requests/test.py::test_combination2 PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[default-] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[nopass-] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[pass-foo] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-1] PASSED test_http_handlers_config/test.py::test_fixed_static_handler PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-0] PASSED test_hedged_requests/test.py::test_combination3 PASSED test_globs_in_filepath/test.py::test_deep_structure PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-1] PASSED test_dotnet_client/test.py::test_dotnet_client PASSED test_keeper_feature_flags_config/test.py::test_keeper_feature_flags PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[0] PASSED test_globs_in_filepath/test.py::test_linear_structure PASSED test_hedged_requests/test.py::test_combination4 PASSED test_http_handlers_config/test.py::test_predefined_query_handler PASSED test_globs_in_filepath/test.py::test_strange_filenames PASSED test_globs_in_filepath/test.py::test_table_function_and_virtual_columns PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[1] PASSED test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4 PASSED test_hedged_requests/test.py::test_initial_receive_timeout PASSED test_http_handlers_config/test.py::test_prometheus_handler PASSED test_hedged_requests/test.py::test_long_query PASSED test_http_handlers_config/test.py::test_relative_path_static_handler PASSED test_executable_user_defined_functions_config_reload/test.py::test PASSED test_hedged_requests/test.py::test_receive_timeout1 PASSED test_hedged_requests/test.py::test_receive_timeout2 PASSED test_endpoint_macro_substitution/test.py::test_different_types PASSED test_http_handlers_config/test.py::test_replicas_status_handler PASSED test_endpoint_macro_substitution/test.py::test_select_by_type PASSED test_hedged_requests/test.py::test_send_data PASSED test_hedged_requests/test.py::test_send_data2 PASSED test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4_fails_with_wrong_resolution PASSED test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total PASSED test_hedged_requests/test.py::test_send_table_status_sleep PASSED test_distributed_load_balancing/test.py::test_load_balancing_first_or_random PASSED test_hedged_requests/test.py::test_send_table_status_sleep2 PASSED test_hedged_requests/test.py::test_stuck_replica PASSED test_distributed_load_balancing/test.py::test_load_balancing_in_order PASSED test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v6 PASSED test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname SKIPPED [1] test_jemalloc_percpu_arena/test.py:62: Compiled without jemalloc (USE_JEMALLOC=b'') ============= 5 failed, 94 passed, 1 skipped in 391.99s (0:06:31) ============== Traceback (most recent call last): File "/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration/./runner", line 455, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_0qpkhw --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_BASE_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_KERBERIZED_HADOOP_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_KERBEROS_KDC_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_MYSQL_JAVA_CLIENT_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_MYSQL_JS_CLIENT_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_MYSQL_PHP_CLIENT_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_NGINX_DAV_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=0-b53fc9f68f30219552c427c7d7d451342728b9a4 -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_NEW_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo]' test_distributed_inter_server_secret/test.py::test_secure test_distributed_inter_server_secret/test.py::test_secure_disagree test_distributed_inter_server_secret/test.py::test_secure_disagree_insert test_distributed_inter_server_secret/test.py::test_secure_insert_async test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async test_distributed_inter_server_secret/test.py::test_secure_insert_sync 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[default-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_from_backward[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[default-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster_with_backward[pass-foo]' test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority]' 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative]' test_distributed_load_balancing/test.py::test_load_balancing_round_robin test_dotnet_client/test.py::test_dotnet_client test_encrypted_disk_replication/test.py::test_replication test_encrypted_disk_replication/test.py::test_zero_copy_replication test_endpoint_macro_substitution/test.py::test_different_types test_endpoint_macro_substitution/test.py::test_select_by_type test_executable_user_defined_functions_config_reload/test.py::test test_globs_in_filepath/test.py::test_deep_structure test_globs_in_filepath/test.py::test_linear_structure test_globs_in_filepath/test.py::test_strange_filenames test_globs_in_filepath/test.py::test_table_function_and_virtual_columns test_graphite_merge_tree/test.py::test_broken_partial_rollup test_graphite_merge_tree/test.py::test_combined_rules test_graphite_merge_tree/test.py::test_combined_rules_with_default test_graphite_merge_tree/test.py::test_multiple_output_blocks test_graphite_merge_tree/test.py::test_multiple_paths_and_versions test_graphite_merge_tree/test.py::test_path_dangling_pointer test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern test_graphite_merge_tree/test.py::test_rollup_aggregation test_graphite_merge_tree/test.py::test_rollup_aggregation_2 test_graphite_merge_tree/test.py::test_rollup_versions test_graphite_merge_tree/test.py::test_system_graphite_retentions test_graphite_merge_tree/test.py::test_wrong_rollup_config test_grpc_protocol_ssl/test.py::test_insecure_channel test_grpc_protocol_ssl/test.py::test_secure_channel test_grpc_protocol_ssl/test.py::test_wrong_client_certificate test_hedged_requests/test.py::test_async_connect test_hedged_requests/test.py::test_async_query_sending test_hedged_requests/test.py::test_combination1 test_hedged_requests/test.py::test_combination2 test_hedged_requests/test.py::test_combination3 test_hedged_requests/test.py::test_combination4 test_hedged_requests/test.py::test_initial_receive_timeout test_hedged_requests/test.py::test_long_query test_hedged_requests/test.py::test_receive_timeout1 test_hedged_requests/test.py::test_receive_timeout2 test_hedged_requests/test.py::test_send_data test_hedged_requests/test.py::test_send_data2 test_hedged_requests/test.py::test_send_table_status_sleep test_hedged_requests/test.py::test_send_table_status_sleep2 test_hedged_requests/test.py::test_stuck_replica test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4 test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v4_fails_with_wrong_resolution test_host_regexp_multiple_ptr_records/test.py::test_host_regexp_multiple_ptr_v6 test_http_handlers_config/test.py::test_absolute_path_static_handler test_http_handlers_config/test.py::test_config_static_handler test_http_handlers_config/test.py::test_defaults_http_handlers test_http_handlers_config/test.py::test_dynamic_query_handler test_http_handlers_config/test.py::test_fixed_static_handler test_http_handlers_config/test.py::test_predefined_query_handler test_http_handlers_config/test.py::test_prometheus_handler test_http_handlers_config/test.py::test_relative_path_static_handler test_http_handlers_config/test.py::test_replicas_status_handler test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_small[1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_different_header[1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_success[1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[0-1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_1[1-1]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[0]' 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_truncated_2[1]' test_jemalloc_percpu_arena/test.py::test_jemalloc_percpu_arena test_keeper_feature_flags_config/test.py::test_keeper_feature_flags -vvv" altinityinfra/integration-tests-runner:0-b53fc9f68f30219552c427c7d7d451342728b9a4 ' returned non-zero exit status 1.